Skip to contentSkip to conversation

erofs-utils: introduce fuse implementation

PATCH Superseded Gao Xiang via Linux-erofs 2 participants 18 messages Oct 17, 2020 – Oct 19, 2020 1 message not in the archive Kbuild

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

cover erofs-utils: introduce fuse implementation

Hi all,

This patchset is based on the original patch
https://lore.kernel.org/r/[email protected]
and several fixes / compression support by Huang Jianan
https://lore.kernel.org/r/[email protected]

with my additional 9 incremental patches to refactor my current development status, which will be folded into original patches in the next WIP version since the fuse approach hasn't been merged into dev branch yet.

To summarize the benefits of erofsfuse, I think it would be:

- erofs images can be supported on various platforms;
- an independent unpack tool can be developed based on this;
- new on-disk feature can be iterated, verified effectively.

Any feedback or/and follow-on development/cleanup is welcomed. (we still have a lot to do for the entire erofsfuse codebase...)

Thanks,
Gao Xiang

Gao Xiang (9):
erofs-utils: fuse: adjust larger extent handling
erofs-utils: fuse: use proper expression about inode size
erofs-utils: fuse: drop ofs_out
erofs-utils: fuse: refuse a undefined shifted cluster behavior
erofs-utils: fuse: drop z_erofs_shifted_transform()
erofs-utils: fuse: rename ofs_head and outputsize
erofs-utils: fuse: cleanup erofs_read_data_compression()
erofs-utils: fuse: move up mpage in struct erofs_map_blocks
erofs-utils: fuse: fix up source headers

Huang Jianan (2):
erofs-utils: fuse: support read special file
erofs-utils: fuse: support read compressed file

Li Guifu (1):
erofs-utils: introduce fuse implementation

 Makefile.am              |   2 +-
 README                   |  28 ++-
 configure.ac             |   3 +-
 fuse/Makefile.am         |  18 ++
 fuse/decompress.c        |  84 ++++++++
 fuse/decompress.h        |  42 ++++
 fuse/dentry.c            | 129 ++++++++++++
 fuse/dentry.h            |  43 ++++
 fuse/disk_io.c           |  72 +++++++
 fuse/disk_io.h           |  21 ++
 fuse/getattr.c           |  65 ++++++
 fuse/getattr.h           |  15 ++
 fuse/init.c              | 118 +++++++++++
 fuse/init.h              |  24 +++
 fuse/logging.c           |  81 ++++++++
 fuse/logging.h           |  55 ++++++
 fuse/main.c              | 171 ++++++++++++++++
 fuse/namei.c             | 242 +++++++++++++++++++++++
 fuse/namei.h             |  22 +++
 fuse/open.c              |  22 +++
 fuse/open.h              |  15 ++
 fuse/read.c              | 214 ++++++++++++++++++++
 fuse/read.h              |  17 ++
 fuse/readir.c            | 123 ++++++++++++
 fuse/readir.h            |  17 ++
 fuse/zmap.c              | 417 +++++++++++++++++++++++++++++++++++++++
 include/erofs/defs.h     |  16 ++
 include/erofs/internal.h |  79 ++++++++
 include/erofs_fs.h       |   4 +
 29 files changed, 2156 insertions(+), 3 deletions(-)
 create mode 100644 fuse/Makefile.am
 create mode 100644 fuse/decompress.c
 create mode 100644 fuse/decompress.h
 create mode 100644 fuse/dentry.c
 create mode 100644 fuse/dentry.h
 create mode 100644 fuse/disk_io.c
 create mode 100644 fuse/disk_io.h
 create mode 100644 fuse/getattr.c
 create mode 100644 fuse/getattr.h
 create mode 100644 fuse/init.c
 create mode 100644 fuse/init.h
 create mode 100644 fuse/logging.c
 create mode 100644 fuse/logging.h
 create mode 100644 fuse/main.c
 create mode 100644 fuse/namei.c
 create mode 100644 fuse/namei.h
 create mode 100644 fuse/open.c
 create mode 100644 fuse/open.h
 create mode 100644 fuse/read.c
 create mode 100644 fuse/read.h
 create mode 100644 fuse/readir.c
 create mode 100644 fuse/readir.h
 create mode 100644 fuse/zmap.c
Signature · 1 line
2.24.0
#2 lore raw

PATCH 1/12 erofs-utils: introduce fuse implementation

From: Li Guifu <[email protected]>

Let's add erofsfuse approach, and benefits are:

- images can be supported on various platforms;
- new unpack tool can be developed based on this;
- new on-disk feature can be iterated, verified effectively.

This commit only aims at reading a regular file. Other file (e.g. compressed file) support is out of scope for now.

---
 Makefile.am              |   2 +-
 README                   |  28 ++++-
 configure.ac             |   3 +-
 fuse/Makefile.am         |  14 +++
 fuse/dentry.c            | 129 ++++++++++++++++++++++
 fuse/dentry.h            |  42 ++++++++
 fuse/disk_io.c           |  72 +++++++++++++
 fuse/disk_io.h           |  21 ++++
 fuse/getattr.c           |  64 +++++++++++
 fuse/getattr.h           |  15 +++
 fuse/init.c              |  98 +++++++++++++++++
 fuse/init.h              |  22 ++++
 fuse/logging.c           |  81 ++++++++++++++
 fuse/logging.h           |  55 ++++++++++
 fuse/main.c              | 170 +++++++++++++++++++++++++++++
 fuse/namei.c             | 227 +++++++++++++++++++++++++++++++++++++++
 fuse/namei.h             |  22 ++++
 fuse/open.c              |  22 ++++
 fuse/open.h              |  15 +++
 fuse/read.c              | 114 ++++++++++++++++++++
 fuse/read.h              |  16 +++
 fuse/readir.c            | 123 +++++++++++++++++++++
 fuse/readir.h            |  17 +++
 include/erofs/defs.h     |   3 +
 include/erofs/internal.h |  37 +++++++
 25 files changed, 1409 insertions(+), 3 deletions(-)
 create mode 100644 fuse/Makefile.am
 create mode 100644 fuse/dentry.c
 create mode 100644 fuse/dentry.h
 create mode 100644 fuse/disk_io.c
 create mode 100644 fuse/disk_io.h
 create mode 100644 fuse/getattr.c
 create mode 100644 fuse/getattr.h
 create mode 100644 fuse/init.c
 create mode 100644 fuse/init.h
 create mode 100644 fuse/logging.c
 create mode 100644 fuse/logging.h
 create mode 100644 fuse/main.c
 create mode 100644 fuse/namei.c
 create mode 100644 fuse/namei.h
 create mode 100644 fuse/open.c
 create mode 100644 fuse/open.h
 create mode 100644 fuse/read.c
 create mode 100644 fuse/read.h
 create mode 100644 fuse/readir.c
 create mode 100644 fuse/readir.h
Makefile.am +1 -1
index 1d20577068c5..24f4a7b3d5ad 100644
@@ -3,4 +3,4 @@
 
 ACLOCAL_AMFLAGS = -I m4
 
-SUBDIRS = man lib mkfs
+SUBDIRS = man lib mkfs fuse
README +27 -1
index 5addd6b80e04..870858c48b7d 100644
@@ -2,7 +2,8 @@ erofs-utils
 ===========
 
 erofs-utils includes user-space tools for erofs filesystem images.
-Currently only mkfs.erofs is available.
+One is mkfs.erofs to create a image, the other is erofsfuse which
+is used to mount a image on a directory
 
 mkfs.erofs
 ----------
@@ -95,6 +96,31 @@ It may still be useful since new erofs-utils has not been widely used in
 commercial products. However, if that happens, please report bug to us
 as well.
 
+erofs-utils: erofsfuse
+----------------------
+erofsfuse mount a erofs filesystem image created by mkfs.erofs to a directory, and then
+It can be listed, read files and so on.
+
+Dependencies
+~~~~~~~~~~~~
+FUSE library version: 2.9.7
+fusermount version: 2.9.7
+using FUSE kernel interface version 7.19
+
+How to installed fuse
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+sudo apt-get update
+sudo apt-get install -y fuse libfuse-dev
+
+How to build erofsfuse
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+	$ ./autogen.sh
+	$ ./configure
+	$ make
+
+erofsfuse binary will be generated under fuse folder.
+
 Contribution
 ------------
 
configure.ac +2 -1
index 0f40a840cf4f..4194a77f1e36 100644
@@ -245,6 +245,7 @@ fi
 AC_CONFIG_FILES([Makefile
 		 man/Makefile
 		 lib/Makefile
-		 mkfs/Makefile])
+		 mkfs/Makefile
+		 fuse/Makefile])
 AC_OUTPUT
 
fuse/Makefile.am new file +14 -0
new file mode 100644
index 000000000000..fffd67a53fe1
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0+
+# Makefile.am
+
+AUTOMAKE_OPTIONS = foreign
+bin_PROGRAMS     = erofsfuse
+erofsfuse_SOURCES = main.c dentry.c getattr.c logging.c namei.c read.c disk_io.c init.c open.c readir.c
+erofsfuse_CFLAGS = -Wall -Werror -Wextra \
+                   -I$(top_srcdir)/include \
+                   $(shell pkg-config fuse --cflags) \
+                   -DFUSE_USE_VERSION=26 \
+                   -std=gnu99
+LDFLAGS += $(shell pkg-config fuse --libs)
+erofsfuse_LDADD = $(top_builddir)/lib/liberofs.la -ldl
+
fuse/dentry.c new file +129 -0

129 lines not on this page · raw message

fuse/dentry.h new file +42 -0

42 lines not on this page · raw message

fuse/disk_io.c new file +72 -0

72 lines not on this page · raw message

fuse/disk_io.h new file +21 -0
new file mode 100644
index 000000000000..6b4bd3cce085
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\disk_io.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __DISK_IO_H
+#define __DISK_IO_H
+
+#include "erofs/defs.h"
+#include "erofs/internal.h"
+
+int dev_open(const char *path);
+void dev_close(void);
+int dev_read(void *buf, size_t count, off_t offset);
+
+static inline int dev_read_blk(void *buf, uint32_t nr)
+{
+	return dev_read(buf, EROFS_BLKSIZ, blknr_to_addr(nr));
+}
+#endif
fuse/getattr.c new file +64 -0

64 lines not on this page · raw message

fuse/getattr.h new file +15 -0
new file mode 100644
index 000000000000..dbcff7c1a6e1
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\getattr.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __EROFS_GETATTR_H
+#define __EROFS_GETATTR_H
+
+#include <fuse.h>
+#include <fuse_opt.h>
+
+int erofs_getattr(const char *path, struct stat *st);
+
+#endif
fuse/init.c new file +98 -0

98 lines not on this page · raw message

fuse/init.h new file +22 -0
new file mode 100644
index 000000000000..d7a97b5ee043
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\init.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __EROFS_INIT_H
+#define __EROFS_INIT_H
+
+#include <fuse.h>
+#include <fuse_opt.h>
+#include "erofs/internal.h"
+
+#define BOOT_SECTOR_SIZE	0x400
+
+int erofs_init_super(void);
+erofs_nid_t erofs_get_root_nid(void);
+erofs_off_t nid2addr(erofs_nid_t nid);
+erofs_nid_t addr2nid(erofs_off_t addr);
+void *erofs_init(struct fuse_conn_info *info);
+
+#endif
fuse/logging.c new file +81 -0

81 lines not on this page · raw message

fuse/logging.h new file +55 -0

55 lines not on this page · raw message

fuse/main.c new file +170 -0

170 lines not on this page · raw message

fuse/namei.c new file +227 -0

227 lines not on this page · raw message

fuse/namei.h new file +22 -0
new file mode 100644
index 000000000000..80e84d7220aa
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\inode.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __INODE_H
+#define __INODE_H
+
+#include "erofs/internal.h"
+#include "erofs_fs.h"
+
+int inode_init(erofs_nid_t root);
+struct dcache_entry *get_cached_dentry(struct dcache_entry **parent,
+				       const char **path);
+int erofs_iget_by_path(const char *path, struct erofs_vnode *v);
+int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *v);
+struct dcache_entry *disk_lookup(struct dcache_entry *parent, const char *name,
+		unsigned int name_len);
+int walk_path(const char *path, erofs_nid_t *out_nid);
+
+#endif
fuse/open.c new file +22 -0
new file mode 100644
index 000000000000..9d2edca54d23
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * erofs-fuse\open.c
+ * Created by Li Guifu <[email protected]>
+ */
+
+#include "open.h"
+#include <asm-generic/errno-base.h>
+#include <fuse.h>
+#include <fuse_opt.h>
+#include "logging.h"
+
+int erofs_open(const char *path, struct fuse_file_info *fi)
+{
+	logi("open path=%s", path);
+
+	if ((fi->flags & O_ACCMODE) != O_RDONLY)
+		return -EACCES;
+
+	return 0;
+}
+
fuse/open.h new file +15 -0
new file mode 100644
index 000000000000..d02c1f752204
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\open.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __EROFS_OPEN_H
+#define __EROFS_OPEN_H
+
+#include <fuse.h>
+#include <fuse_opt.h>
+
+int erofs_open(const char *path, struct fuse_file_info *fi);
+
+#endif
fuse/read.c new file +114 -0

114 lines not on this page · raw message

fuse/read.h new file +16 -0
new file mode 100644
index 000000000000..f2fcebec9d07
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\read.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __EROFS_READ_H
+#define __EROFS_READ_H
+
+#include <fuse.h>
+#include <fuse_opt.h>
+
+int erofs_read(const char *path, char *buffer, size_t size, off_t offset,
+	    struct fuse_file_info *fi);
+
+#endif
fuse/readir.c new file +123 -0

123 lines not on this page · raw message

fuse/readir.h new file +17 -0
new file mode 100644
index 000000000000..21ab7a4e4e0b
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * erofs-fuse\readir.h
+ * Created by Li Guifu <[email protected]>
+ */
+
+#ifndef __EROFS_READDIR_H
+#define __EROFS_READDIR_H
+
+#include <fuse.h>
+#include <fuse_opt.h>
+
+int erofs_readdir(const char *path, void *buffer, fuse_fill_dir_t filler,
+	       off_t offset, struct fuse_file_info *fi);
+
+
+#endif
include/erofs/defs.h +3 -0
index 33320dbe8e3c..a9c769ec88ae 100644
@@ -24,6 +24,9 @@
 #ifdef HAVE_LINUX_TYPES_H
 #include <linux/types.h>
 #endif
+#ifndef UNUSED
+#define UNUSED(_X)    ((void) _X)
+#endif
 
 /*
  * container_of - cast a member of a structure out to the containing structure
include/erofs/internal.h +37 -0
index bc77c43719e8..cba3ce4695d4 100644
@@ -55,6 +55,8 @@ typedef u32 erofs_blk_t;
 #define blknr_to_addr(nr)       ((erofs_off_t)(nr) * EROFS_BLKSIZ)
 
 #define BLK_ROUND_UP(addr)	DIV_ROUND_UP(addr, EROFS_BLKSIZ)
+#define IS_SLOT_ALIGN(__ADDR)   (((__ADDR) % (EROFS_SLOTSIZE)) ? 0 : 1)
+#define IS_BLK_ALIGN(__ADDR)    (((__ADDR) % (EROFS_BLKSIZ)) ? 0 : 1)
 
 struct erofs_buffer_head;
 
@@ -132,11 +134,46 @@ struct erofs_inode {
 #endif
 };
 
+struct erofs_vnode {
+	uint8_t datalayout;
+
+	uint32_t i_size;
+	/* inline size in bytes */
+	uint16_t inode_isize;
+	uint16_t xattr_isize;
+
+	uint16_t xattr_shared_count;
+	char *xattr_shared_xattrs;
+
+	erofs_blk_t raw_blkaddr;
+	erofs_nid_t nid;
+	uint32_t i_ino;
+
+	uint16_t i_mode;
+	uint16_t i_uid;
+	uint16_t i_gid;
+	uint16_t i_nlink;
+
+	/* if file is inline read inline data witch inode */
+	char *idata;
+};
+
 static inline bool is_inode_layout_compression(struct erofs_inode *inode)
 {
 	return erofs_inode_is_data_compressed(inode->datalayout);
 }
 
+#define __inode_advise(x, bit, bits) \
+		(((x) >> (bit)) & ((1 << (bits)) - 1))
+
+#define __inode_version(advise)	\
+		__inode_advise(advise, EROFS_I_VERSION_BIT,	\
+			EROFS_I_VERSION_BITS)
+
+#define __inode_data_mapping(advise)	\
+	__inode_advise(advise, EROFS_I_DATALAYOUT_BIT,\
+		EROFS_I_DATALAYOUT_BITS)
+
 #define IS_ROOT(x)	((x) == (x)->i_parent)
 
 struct erofs_dentry {
Signature · 1 line
2.24.0
#3 lore raw

PATCH 2/12 erofs-utils: fuse: support read special file

In reply to Gao Xiang via Linux-erofs

From: Huang Jianan <[email protected]>

---
 fuse/getattr.c           |  1 +
 fuse/main.c              |  1 +
 fuse/namei.c             | 14 ++++++++++----
 fuse/read.c              | 26 ++++++++++++++++++++++++++
 fuse/read.h              |  1 +
 include/erofs/internal.h |  1 +
 6 files changed, 40 insertions(+), 4 deletions(-)
fuse/getattr.c +1 -0
index 542cf35f6989..d2134f486e19 100644
@@ -56,6 +56,7 @@ int erofs_getattr(const char *path, struct stat *stbuf)
 	stbuf->st_blocks = stbuf->st_size / EROFS_BLKSIZ;
 	stbuf->st_uid = le16_to_cpu(v.i_uid);
 	stbuf->st_gid = le16_to_cpu(v.i_gid);
+	stbuf->st_rdev = le32_to_cpu(v.i_rdev);
 	stbuf->st_atime = super.build_time;
 	stbuf->st_mtime = super.build_time;
 	stbuf->st_ctime = super.build_time;
fuse/main.c +1 -0
index fa9795ef1615..884101374bcf 100644
@@ -113,6 +113,7 @@ static void signal_handle_sigsegv(int signal)
 }
 
 static struct fuse_operations erofs_ops = {
+	.readlink = erofs_readlink,
 	.getattr = erofs_getattr,
 	.readdir = erofs_readdir,
 	.open = erofs_open,
fuse/namei.c +10 -4
index 3503a8d56e15..7ed1168fc14f 100644
@@ -11,6 +11,7 @@
 #include <stdio.h>
 #include <errno.h>
 #include <sys/stat.h>
+#include <sys/sysmacros.h>
 
 #include "erofs/defs.h"
 #include "logging.h"
@@ -39,6 +40,13 @@ static uint8_t get_path_token_len(const char *path)
 	return len;
 }
 
+static inline dev_t new_decode_dev(u32 dev)
+{
+	unsigned major = (dev & 0xfff00) >> 8;
+	unsigned minor = (dev & 0xff) | ((dev >> 12) & 0xfff00);
+	return makedev(major, minor);
+}
+
 int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
 {
 	int ret;
@@ -65,13 +73,11 @@ int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
 	switch (vi->i_mode & S_IFMT) {
 	case S_IFBLK:
 	case S_IFCHR:
-		/* fixme: add special devices support
-		 * vi->i_rdev = new_decode_dev(le32_to_cpu(v1->i_u.rdev));
-		 */
+		vi->i_rdev = new_decode_dev(le32_to_cpu(v1->i_u.rdev));
 		break;
 	case S_IFIFO:
 	case S_IFSOCK:
-		/*fixme: vi->i_rdev = 0; */
+		vi->i_rdev = 0;
 		break;
 	case S_IFREG:
 	case S_IFLNK:
fuse/read.c +26 -0
index ffe976e59a11..3ce5c4f2911e 100644
@@ -112,3 +112,29 @@ int erofs_read(const char *path, char *buffer, size_t size, off_t offset,
 		return -EINVAL;
 	}
 }
+
+int erofs_readlink(const char *path, char *buffer, size_t size)
+{
+	int ret;
+	erofs_nid_t nid;
+	size_t lnksz;
+	struct erofs_vnode v;
+
+	ret = walk_path(path, &nid);
+	if (ret)
+		return ret;
+
+	ret = erofs_iget_by_nid(nid, &v);
+	if (ret)
+		return ret;
+
+	lnksz = min((size_t)v.i_size, size - 1);
+
+	ret = erofs_read(path, buffer, lnksz, 0, NULL);
+	buffer[lnksz] = '\0';
+	if (ret != (int)lnksz)
+		return ret;
+
+	logi("path:%s link=%s size=%llu", path, buffer, lnksz);
+	return 0;
+}
\ No newline at end of file
fuse/read.h +1 -0
index f2fcebec9d07..89d4b4cd600c 100644
@@ -12,5 +12,6 @@
 
 int erofs_read(const char *path, char *buffer, size_t size, off_t offset,
 	    struct fuse_file_info *fi);
+int erofs_readlink(const char *path, char *buffer, size_t size);
 
 #endif
include/erofs/internal.h +1 -0
index cba3ce4695d4..47ad96d0488c 100644
@@ -153,6 +153,7 @@ struct erofs_vnode {
 	uint16_t i_uid;
 	uint16_t i_gid;
 	uint16_t i_nlink;
+	uint32_t i_rdev;
 
 	/* if file is inline read inline data witch inode */
 	char *idata;
Signature · 1 line
2.24.0
#4 lore raw

PATCH 3/12 erofs-utils: fuse: support read compressed file

In reply to Gao Xiang via Linux-erofs

From: Huang Jianan <[email protected]>

---
 fuse/Makefile.am         |   8 +-
 fuse/decompress.c        |  86 ++++++++
 fuse/decompress.h        |  37 ++++
 fuse/dentry.h            |   5 +-
 fuse/init.c              |  22 ++-
 fuse/init.h              |   2 +
 fuse/namei.c             |  11 +-
 fuse/read.c              |  73 ++++++-
 fuse/zmap.c              | 416 +++++++++++++++++++++++++++++++++++++++
 include/erofs/defs.h     |  13 ++
 include/erofs/internal.h |  43 +++-
 include/erofs_fs.h       |   4 +
 12 files changed, 711 insertions(+), 9 deletions(-)
 create mode 100644 fuse/decompress.c
 create mode 100644 fuse/decompress.h
 create mode 100644 fuse/zmap.c
fuse/Makefile.am +6 -2
index fffd67a53fe1..052c7163dff7 100644
@@ -3,12 +3,16 @@
 
 AUTOMAKE_OPTIONS = foreign
 bin_PROGRAMS     = erofsfuse
-erofsfuse_SOURCES = main.c dentry.c getattr.c logging.c namei.c read.c disk_io.c init.c open.c readir.c
+erofsfuse_SOURCES = main.c dentry.c getattr.c logging.c namei.c read.c disk_io.c init.c open.c readir.c zmap.c
+if ENABLE_LZ4
+erofsfuse_SOURCES += decompress.c
+endif
 erofsfuse_CFLAGS = -Wall -Werror -Wextra \
+                   -Wno-implicit-fallthrough \
                    -I$(top_srcdir)/include \
                    $(shell pkg-config fuse --cflags) \
                    -DFUSE_USE_VERSION=26 \
                    -std=gnu99
 LDFLAGS += $(shell pkg-config fuse --libs)
-erofsfuse_LDADD = $(top_builddir)/lib/liberofs.la -ldl
+erofsfuse_LDADD = $(top_builddir)/lib/liberofs.la -ldl ${LZ4_LIBS}
 
fuse/decompress.c new file +86 -0

86 lines not on this page · raw message

fuse/decompress.h new file +37 -0
new file mode 100644
index 000000000000..cd395c3d6b3f
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (C), 2008-2020, OPPO Mobile Comm Corp., Ltd.
+ * Created by Huang Jianan <[email protected]>
+ */
+
+#ifndef __EROFS_DECOMPRESS_H
+#define __EROFS_DECOMPRESS_H
+
+#include "erofs/internal.h"
+
+enum {
+	Z_EROFS_COMPRESSION_SHIFTED = Z_EROFS_COMPRESSION_MAX,
+	Z_EROFS_COMPRESSION_RUNTIME_MAX
+};
+
+struct z_erofs_decompress_req {
+	char *in, *out;
+
+	size_t ofs_out, ofs_head;
+	unsigned int inputsize, outputsize;
+
+	/* indicate the algorithm will be used for decompression */
+	unsigned int alg;
+	bool partial_decoding;
+};
+
+#ifdef LZ4_ENABLED
+int z_erofs_decompress(struct z_erofs_decompress_req *rq);
+#else
+int z_erofs_decompress(struct z_erofs_decompress_req *rq)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
+#endif
fuse/dentry.h +3 -2
index ee2144de8a89..f89c50646fb5 100644
@@ -10,10 +10,11 @@
 #include <stdint.h>
 #include "erofs/internal.h"
 
+/* fixme: Deal with names that exceed the allocated size */
 #ifdef __64BITS
-#define DCACHE_ENTRY_NAME_LEN       40
+#define DCACHE_ENTRY_NAME_LEN       EROFS_NAME_LEN
 #else
-#define DCACHE_ENTRY_NAME_LEN       48
+#define DCACHE_ENTRY_NAME_LEN       EROFS_NAME_LEN
 #endif
 
 /* This struct declares a node of a k-tree.  Every node has a pointer to one of
fuse/init.c +21 -1
index 8198fa78da96..e9cc9f81d4c7 100644
@@ -17,7 +17,23 @@
 
 
 struct erofs_super_block super;
-static struct erofs_super_block *sbk = &super;
+struct erofs_super_block *sbk = &super;
+
+static bool check_layout_compatibility(struct erofs_super_block *sb,
+				       struct erofs_super_block *dsb)
+{
+	const unsigned int feature = le32_to_cpu(dsb->feature_incompat);
+
+	sb->feature_incompat = feature;
+
+	/* check if current kernel meets all mandatory requirements */
+	if (feature & (~EROFS_ALL_FEATURE_INCOMPAT)) {
+		loge("unidentified incompatible feature %x, please upgrade kernel version",
+		     feature & ~EROFS_ALL_FEATURE_INCOMPAT);
+		return false;
+	}
+	return true;
+}
 
 int erofs_init_super(void)
 {
@@ -40,6 +56,9 @@ int erofs_init_super(void)
 		return -EINVAL;
 	}
 
+	if (!check_layout_compatibility(sbk, sb))
+		return -EINVAL;
+
 	sbk->checksum = le32_to_cpu(sb->checksum);
 	sbk->feature_compat = le32_to_cpu(sb->feature_compat);
 	sbk->blkszbits = sb->blkszbits;
@@ -56,6 +75,7 @@ int erofs_init_super(void)
 	sbk->root_nid = le16_to_cpu(sb->root_nid);
 
 	logp("%-15s:0x%X", STR(magic), SUPER_MEM(magic));
+	logp("%-15s:0x%X", STR(feature_incompat), SUPER_MEM(feature_incompat));
 	logp("%-15s:0x%X", STR(feature_compat), SUPER_MEM(feature_compat));
 	logp("%-15s:%u",   STR(blkszbits), SUPER_MEM(blkszbits));
 	logp("%-15s:%u",   STR(root_nid), SUPER_MEM(root_nid));
fuse/init.h +2 -0
index d7a97b5ee043..3fc4eb548dda 100644
@@ -13,6 +13,8 @@
 
 #define BOOT_SECTOR_SIZE	0x400
 
+extern struct erofs_super_block *sbk;
+
 int erofs_init_super(void);
 erofs_nid_t erofs_get_root_nid(void);
 erofs_off_t nid2addr(erofs_nid_t nid);
fuse/namei.c +10 -1
index 7ed1168fc14f..510fcfda5a76 100644
@@ -49,7 +49,7 @@ static inline dev_t new_decode_dev(u32 dev)
 
 int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
 {
-	int ret;
+	int ret, ifmt;
 	char buf[EROFS_BLKSIZ];
 	struct erofs_inode_compact *v1;
 	const erofs_off_t addr = nid2addr(nid);
@@ -60,6 +60,11 @@ int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
 		return -EIO;
 
 	v1 = (struct erofs_inode_compact *)buf;
+	/* fixme: support extended inode */
+	ifmt = le16_to_cpu(v1->i_format);
+	if (__inode_version(ifmt) != EROFS_INODE_LAYOUT_COMPACT)
+		return -EOPNOTSUPP;
+
 	vi->datalayout = __inode_data_mapping(le16_to_cpu(v1->i_format));
 	vi->inode_isize = sizeof(struct erofs_inode_compact);
 	vi->xattr_isize = erofs_xattr_ibody_size(v1->i_xattr_icount);
@@ -88,6 +93,10 @@ int erofs_iget_by_nid(erofs_nid_t nid, struct erofs_vnode *vi)
 		return -EIO;
 	}
 
+	vi->z_inited = false;
+	if (erofs_inode_is_data_compressed(vi->datalayout))
+		z_erofs_fill_inode(vi);
+
 	return 0;
 }
 
fuse/read.c +71 -2

95 lines not on this page · raw message

fuse/zmap.c new file +416 -0

416 lines not on this page · raw message

include/erofs/defs.h +13 -0
index a9c769ec88ae..06d29a9575da 100644
@@ -173,5 +173,18 @@ typedef int64_t         s64;
 #define __maybe_unused      __attribute__((__unused__))
 #endif
 
+struct __una_u32 { u32 x; } __packed;
+
+static inline u32 __get_unaligned_cpu32(const void *p)
+{
+	const struct __una_u32 *ptr = (const struct __una_u32 *)p;
+	return ptr->x;
+}
+
+static inline u32 get_unaligned_le32(const void *p)
+{
+	return __get_unaligned_cpu32((const u8 *)p);
+}
+
 #endif
 
include/erofs/internal.h +42 -1

66 lines not on this page · raw message

include/erofs_fs.h +4 -0
index bcc4f0c630ad..0c70897d2113 100644
@@ -279,6 +279,10 @@ struct z_erofs_vle_decompressed_index {
 	} di_u;
 };
 
+#define Z_EROFS_VLE_LEGACY_INDEX_ALIGN(size) \
+	(round_up(size, sizeof(struct z_erofs_vle_decompressed_index)) + \
+	 sizeof(struct z_erofs_map_header) + Z_EROFS_VLE_LEGACY_HEADER_PADDING)
+
 #define Z_EROFS_VLE_EXTENT_ALIGN(size) round_up(size, \
 	sizeof(struct z_erofs_vle_decompressed_index))
 
Signature · 1 line
2.24.0
#5 lore raw

PATCH 4/12 erofs-utils: fuse: adjust larger extent handling

In reply to Gao Xiang via Linux-erofs

so more easy to understand.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>
---
 fuse/read.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
fuse/read.c +9 -4
index 0d0e3b0fa468..dd44adaa1c40 100644
@@ -112,12 +112,17 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 						Z_EROFS_COMPRESSION_LZ4 :
 						Z_EROFS_COMPRESSION_SHIFTED;
 
-		if (end >= map.m_la + map.m_llen) {
-			count = map.m_llen;
-			partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
-		} else {
+		/*
+		 * trim to the needed size if the returned extent is quite
+		 * larger than requested, and set up partial flag as well.
+		 */
+		if (end < map.m_la + map.m_llen) {
 			count = end - map.m_la;
 			partial = true;
+		} else {
+			ASSERT(end == map.m_la + map_m_llen);
+			count = map.m_llen;
+			partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
 		}
 
 		if ((off_t)map.m_la < offset) {
Signature · 1 line
2.24.0
#6 lore raw

回复: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling

Hi, Gao Xiang,
________________________________

发件人: Gao Xiang<mailto:[email protected]>
发送时间: 2020-10-17 13:16
收件人: linux-erofs<mailto:[email protected]>
抄送: Huang Jianan<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Chao Yu<mailto:[email protected]>; Guo Weichao<mailto:[email protected]>; Zhang Shiming<mailto:[email protected]>; Gao Xiang<mailto:[email protected]> 主题: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling
so more easy to understand.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>

---
fuse/read.c | 13 +++++++++----

1 file changed, 9 insertions(+), 4 deletions(-)
fuse/read.c +0 -0
index 0d0e3b0fa468..dd44adaa1c40 100644
@@ -112,12 +112,17 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,

Z_EROFS_COMPRESSION_LZ4 :
Z_EROFS_COMPRESSION_SHIFTED;
- if (end >= map.m_la + map.m_llen) {
- count = map.m_llen;
- partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
- } else {
+ /*
+ * trim to the needed size if the returned extent is quite
+ * larger than requested, and set up partial flag as well.
+ */
+ if (end < map.m_la + map.m_llen) {
count = end - map.m_la;
partial = true;
+ } else {
+ ASSERT(end == map.m_la + map_m_llen);

I think you mean map.m_llen intesad of map_m_llen.
Besides, I don't understand why add ASSERT here.
I think this condition will be true if offset+size is exactly the end of a compressed block?

+ count = map.m_llen;
+ partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED); }
if ((off_t)map.m_la < offset) {
--
2.24.0

Thanks,
Jianan

________________________________
OPPO

本电子邮件及其附件含有OPPO公司的保密信息,仅限于邮件指明的收件人使用(包含个人及群组)。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件,请立即以电子邮件通知发件人并删除本邮件及其附件。

This e-mail and its attachments contain confidential information from OPPO, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!

#7 lore raw

Re: 回复: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling

Hi Jianan,

Quoted 38 lines · On Mon, Oct 19, 2020 at 11:13:42AM +0800, [email protected] wrote:

Hi, Gao Xiang锟斤拷
________________________________

锟斤拷锟斤拷锟剿o拷 Gao Xiang<mailto:[email protected]>
锟斤拷锟斤拷时锟戒: 2020-10-17 13:16
锟秸硷拷锟剿o拷 linux-erofs<mailto:[email protected]>
锟斤拷锟酵o拷 Huang Jianan<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Chao Yu<mailto:[email protected]>; Guo Weichao<mailto:[email protected]>; Zhang Shiming<mailto:[email protected]>; Gao Xiang<mailto:[email protected]> 锟斤拷锟解: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling
so more easy to understand.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>

---
fuse/read.c | 13 +++++++++----

1 file changed, 9 insertions(+), 4 deletions(-)
fuse/read.c +0 -0
index 0d0e3b0fa468..dd44adaa1c40 100644
@@ -112,12 +112,17 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,

Z_EROFS_COMPRESSION_LZ4 :
Z_EROFS_COMPRESSION_SHIFTED;
- if (end >= map.m_la + map.m_llen) {
- count = map.m_llen;
- partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
- } else {
+ /*
+ * trim to the needed size if the returned extent is quite
+ * larger than requested, and set up partial flag as well.
+ */
+ if (end < map.m_la + map.m_llen) {
count = end - map.m_la;
partial = true;
+ } else {
+ ASSERT(end == map.m_la + map_m_llen);

Quoted 3 lines

I think you mean map.m_llen intesad of map_m_llen.
Besides, I don't understand why add ASSERT here.
I think this condition will be true if offset+size is exactly the end of a compressed block?

Thanks for your question.
The idea is that we requested the extent with

	map.m_la = end - 1;

	ret = z_erofs_map_blocks_iter(vnode, &map);
	if (ret)
		return ret;

so the extent must include "end - 1", so it's impossible that "end > map.m_la + map.m_llen" (invalid return).

or the entire extent would be holed extent, anyway, that is another extent rather than a data extent.

(BTW, the up-to-date commits is at
https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/log/?h=wip/experimental_fuse
kindly check out them as well :) )

Thanks,
Gao Xiang

Quoted 10 lines

+ count = map.m_llen;
+ partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED); }
if ((off_t)map.m_la < offset) {
--
2.24.0

Thanks,
Jianan

#8 lore raw

回复: Re: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling

Hi, Gao Xiang,
Thanks for you answer, I understand now.
In addition, I sent a patch to fix the clerical error here.
It seems that the compilation won't be affected under the default configuration.

Thanks,
Jianan
--------------
Huang Jianan

Quoted 61 lines

Hi Jianan,

Quoted 38 lines · On Mon, Oct 19, 2020 at 11:13:42AM +0800, [email protected] wrote:

Hi, Gao Xiang锟斤拷
________________________________

锟斤拷锟斤拷锟剿o拷 Gao Xiang<mailto:[email protected]>
锟斤拷锟斤拷时锟戒: 2020-10-17 13:16
锟秸硷拷锟剿o拷 linux-erofs<mailto:[email protected]>
锟斤拷锟酵o拷 Huang Jianan<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Li Guifu<mailto:[email protected]>; Chao Yu<mailto:[email protected]>; Guo Weichao<mailto:[email protected]>; Zhang Shiming<mailto:[email protected]>; Gao Xiang<mailto:[email protected]> 锟斤拷锟解: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling
so more easy to understand.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>

---
fuse/read.c | 13 +++++++++----

1 file changed, 9 insertions(+), 4 deletions(-)
fuse/read.c +0 -0
index 0d0e3b0fa468..dd44adaa1c40 100644
@@ -112,12 +112,17 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,

Z_EROFS_COMPRESSION_LZ4 :
Z_EROFS_COMPRESSION_SHIFTED;
- if (end >= map.m_la + map.m_llen) {
- count = map.m_llen;
- partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
- } else {
+ /*
+ * trim to the needed size if the returned extent is quite
+ * larger than requested, and set up partial flag as well.
+ */
+ if (end < map.m_la + map.m_llen) {
count = end - map.m_la;
partial = true;
+ } else {
+ ASSERT(end == map.m_la + map_m_llen);

Quoted 3 lines

I think you mean map.m_llen intesad of map_m_llen.
Besides, I don't understand why add ASSERT here.
I think this condition will be true if offset+size is exactly the end of a compressed block?

Thanks for your question.
The idea is that we requested the extent with

      map.m_la = end - 1;

      ret = z_erofs_map_blocks_iter(vnode, &map);
      if (ret)
      return ret;

so the extent must include "end - 1", so it's impossible that "end > map.m_la + map.m_llen" (invalid return).

or the entire extent would be holed extent, anyway, that is another extent rather than a data extent.

Quoted 19 lines

(BTW, the up-to-date commits is at
https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/log/?h=wip/experimental_fuse
kindly check out them as well :) )

Thanks,
Gao Xiang

Quoted 10 lines

+ count = map.m_llen;
+ partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED); }
if ((off_t)map.m_la < offset) {
--
2.24.0

Thanks,
Jianan

________________________________
OPPO

本电子邮件及其附件含有OPPO公司的保密信息,仅限于邮件指明的收件人使用(包含个人及群组)。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件,请立即以电子邮件通知发件人并删除本邮件及其附件。

This e-mail and its attachments contain confidential information from OPPO, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!

#9 lore raw

回复: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling

In reply to Gao Xiang via Linux-erofs

Hi, Gao Xiang,
Sorry for the format of the previous email.

Quoted 28 lines

so more easy to understand.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>

---
fuse/read.c | 13 +++++++++----

1 file changed, 9 insertions(+), 4 deletions(-)
fuse/read.c +0 -0
index 0d0e3b0fa468..dd44adaa1c40 100644
@@ -112,12 +112,17 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,

Z_EROFS_COMPRESSION_LZ4 :
Z_EROFS_COMPRESSION_SHIFTED;

fuse/read.c +5 -4
-      if (end >= map.m_la + map.m_llen) {
-      count = map.m_llen;
-      partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
-      } else {
+      /*
+      * trim to the needed size if the returned extent is quite
+      * larger than requested, and set up partial flag as well.
+      */
+      if (end < map.m_la + map.m_llen) {

count = end - map.m_la;
partial = true;

fuse/read.c size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer, · line 123

Quoted 2 lines
fuse/read.c +2 -0
+      } else {
+      ASSERT(end == map.m_la + map_m_llen);

I think you mean map.m_llen intesad of map_m_llen.
Besides, I don't understand why add ASSERT here.
I think this condition will be true if offset+size is exactly the end of a compressed block?

Quoted 8 lines
fuse/read.c +2 -0
+      count = map.m_llen;
+      partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);

}

if ((off_t)map.m_la < offset) {
--
2.24.0

Thanks,
Jianan
________________________________
OPPO

本电子邮件及其附件含有OPPO公司的保密信息,仅限于邮件指明的收件人使用(包含个人及群组)。禁止任何人在未经授权的情况下以任何形式使用。如果您错收了本邮件,请立即以电子邮件通知发件人并删除本邮件及其附件。

This e-mail and its attachments contain confidential information from OPPO, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!

#10 lore raw

Re: 回复: [WIP] [PATCH 04/12] erofs-utils: fuse: adjust larger extent handling

Hi Jianan,

Quoted 2 lines · Huang Jianan

Hi, Gao Xiang,
Sorry for the format of the previous email.

I've replied the previous email at
https://lore.kernel.org/r/20201019033251.GA29138@hsiangkao-HP-ZHAN-66-Pro-G1

As for the email format, I think you can give a try to use mutt or Thunderbird (assume your company PC is in MS Windows) , which is more widely used in open-source communities :)

Thanks,
Gao Xiang

Quoted 1 line
#11 lore raw

PATCH 5/12 erofs-utils: fuse: use proper expression about inode size

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>
---
 fuse/namei.c  | 4 ++--
 fuse/read.c   | 2 +-
 fuse/readir.c | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
fuse/namei.c +2 -2
index 510fcfda5a76..0b71072027ce 100644
@@ -172,8 +172,8 @@ struct dcache_entry *disk_lookup(struct dcache_entry *parent, const char *name,
 
 	if (v.datalayout == EROFS_INODE_FLAT_INLINE) {
 		uint32_t dir_off = erofs_blkoff(dirsize);
-		off_t dir_addr = nid2addr(dcache_get_nid(parent))
-			+ sizeof(struct erofs_inode_compact) + v.xattr_isize;
+		off_t dir_addr = nid2addr(dcache_get_nid(parent)) +
+			v.inode_isize + v.xattr_isize;
 
 		memset(buf, 0, sizeof(buf));
 		ret = dev_read(buf, dir_off, dir_addr);
fuse/read.c +1 -1
index dd44adaa1c40..3f2e49c390de 100644
@@ -65,7 +65,7 @@ size_t erofs_read_data_inline(struct erofs_vnode *vnode, char *buffer,
 	if (!suminline)
 		goto finished;
 
-	addr = nid2addr(vnode->nid) + sizeof(struct erofs_inode_compact)
+	addr = nid2addr(vnode->nid) + vnode->inode_isize +
 		+ vnode->xattr_isize;
 	ret = dev_read(buffer + rdsz, suminline, addr);
 	if (ret < 0 || (size_t)ret != suminline)
fuse/readir.c +1 -1
index 9589685c9122..46ceb1d90a7f 100644
@@ -108,7 +108,7 @@ int erofs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
 	if (v.datalayout == EROFS_INODE_FLAT_INLINE) {
 		off_t addr;
 
-		addr = nid2addr(nid) + sizeof(struct erofs_inode_compact)
+		addr = nid2addr(nid) + v.inode_isize
 			+ v.xattr_isize;
 
 		memset(dirsbuf, 0, sizeof(dirsbuf));
Signature · 1 line
2.24.0
#12 lore raw

PATCH 6/12 erofs-utils: fuse: drop ofs_out

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>
---
 fuse/decompress.c | 7 +++----
 fuse/decompress.h | 2 +-
 fuse/read.c       | 3 +--
 3 files changed, 5 insertions(+), 7 deletions(-)
fuse/decompress.c +3 -4
index e2df3cea78f7..4b7cf3211319 100644
@@ -15,7 +15,7 @@
 
 static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq)
 {
-	char *dest = rq->out + rq->ofs_out;
+	char *dest = rq->out;
 	char *src = rq->in + rq->ofs_head;
 
 	memcpy(dest, src, rq->outputsize - rq->ofs_head);
@@ -26,7 +26,7 @@ static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq)
 static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
 {
 	int ret = 0;
-	char *dest = rq->out + rq->ofs_out;
+	char *dest = rq->out;
 	char *src = rq->in;
 	char *buff = NULL;
 	bool support_0padding = false;
@@ -66,8 +66,7 @@ static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
 
 	if (rq->ofs_head) {
 		src = dest + rq->ofs_head;
-		dest = rq->out + rq->ofs_out;
-		memcpy(dest, src, rq->outputsize - rq->ofs_head);
+		memcpy(rq->out, src, rq->outputsize - rq->ofs_head);
 	}
 
 out:
fuse/decompress.h +1 -1
index cd395c3d6b3f..5a3e2356dc1b 100644
@@ -17,7 +17,7 @@ enum {
 struct z_erofs_decompress_req {
 	char *in, *out;
 
-	size_t ofs_out, ofs_head;
+	size_t ofs_head;
 	unsigned int inputsize, outputsize;
 
 	/* indicate the algorithm will be used for decompression */
fuse/read.c +1 -2
index 3f2e49c390de..06ca08587e7c 100644
@@ -135,8 +135,7 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 
 		ret = z_erofs_decompress(&(struct z_erofs_decompress_req) {
 					.in = raw,
-					.out = buffer,
-					.ofs_out = sum,
+					.out = buffer + sum,
 					.ofs_head = ofs,
 					.inputsize = EROFS_BLKSIZ,
 					.outputsize = count,
Signature · 1 line
2.24.0
#13 lore raw

PATCH 7/12 erofs-utils: fuse: refuse a undefined shifted cluster behavior

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>
---
 fuse/decompress.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
fuse/decompress.c +4 -1
index 4b7cf3211319..4a9f8e7995c1 100644
@@ -18,8 +18,11 @@ static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq)
 	char *dest = rq->out;
 	char *src = rq->in + rq->ofs_head;
 
-	memcpy(dest, src, rq->outputsize - rq->ofs_head);
+	if (rq->inputsize != EROFS_BLKSIZ)
+		return -EFSCORRUPTED;
 
+	DBG_BUGON(rq->outputsize > EROFS_BLKSIZ);
+	memcpy(dest, src, rq->outputsize - rq->ofs_head);
 	return 0;
 }
 
Signature · 1 line
2.24.0
#14 lore raw

PATCH 8/12 erofs-utils: fuse: drop z_erofs_shifted_transform()

In reply to Gao Xiang via Linux-erofs

clean up as well.

[ let's fold in to the original patch. ]
Cc Huang Jianan <[email protected]>
---
 fuse/decompress.c | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)
fuse/decompress.c +11 -15
index 4a9f8e7995c1..d3ee3677e9a3 100644
@@ -13,19 +13,6 @@
 #include "logging.h"
 #include "init.h"
 
-static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq)
-{
-	char *dest = rq->out;
-	char *src = rq->in + rq->ofs_head;
-
-	if (rq->inputsize != EROFS_BLKSIZ)
-		return -EFSCORRUPTED;
-
-	DBG_BUGON(rq->outputsize > EROFS_BLKSIZ);
-	memcpy(dest, src, rq->outputsize - rq->ofs_head);
-	return 0;
-}
-
 static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
 {
 	int ret = 0;
@@ -81,8 +68,17 @@ out:
 
 int z_erofs_decompress(struct z_erofs_decompress_req *rq)
 {
-	if (rq->alg == Z_EROFS_COMPRESSION_SHIFTED)
-		return z_erofs_shifted_transform(rq);
+	if (rq->alg == Z_EROFS_COMPRESSION_SHIFTED) {
+		if (rq->inputsize != EROFS_BLKSIZ)
+			return -EFSCORRUPTED;
+
+		DBG_BUGON(rq->outputsize > EROFS_BLKSIZ);
+		DBG_BUGON(rq->outputsize < rq->ofs_head);
+
+		memcpy(rq->out, rq->in + rq->ofs_head,
+		       rq->outputsize - rq->ofs_head);
+		return 0;
+	}
 
 	return z_erofs_decompress_generic(rq);
 }
Signature · 1 line
2.24.0
#15 lore raw

PATCH 9/12 erofs-utils: fuse: rename ofs_head and outputsize

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
---
 fuse/decompress.c | 27 +++++++++++++--------------
 fuse/decompress.h |  8 ++++++--
 fuse/read.c       |  4 ++--
 3 files changed, 21 insertions(+), 18 deletions(-)
fuse/decompress.c +13 -14
index d3ee3677e9a3..fc12852ac6b7 100644
@@ -33,8 +33,8 @@ static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
 			return -EIO;
 	}
 
-	if (rq->ofs_head) {
-		buff = malloc(rq->outputsize);
+	if (rq->decodedskip) {
+		buff = malloc(rq->decodedlength);
 		if (!buff)
 			return -ENOMEM;
 		dest = buff;
@@ -42,22 +42,21 @@ static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq)
 
 	if (rq->partial_decoding || !support_0padding)
 		ret = LZ4_decompress_safe_partial(src + inputmargin, dest,
-						  rq->inputsize - inputmargin,
-						  rq->outputsize, rq->outputsize);
+				rq->inputsize - inputmargin,
+				rq->decodedlength, rq->decodedlength);
 	else
 		ret = LZ4_decompress_safe(src + inputmargin, dest,
 					  rq->inputsize - inputmargin,
-					  rq->outputsize);
+					  rq->decodedlength);
 
-	if (ret != (int)rq->outputsize) {
+	if (ret != (int)rq->decodedlength) {
 		ret = -EIO;
 		goto out;
 	}
 
-	if (rq->ofs_head) {
-		src = dest + rq->ofs_head;
-		memcpy(rq->out, src, rq->outputsize - rq->ofs_head);
-	}
+	if (rq->decodedskip)
+		memcpy(rq->out, dest + rq->decodedskip,
+		       rq->decodedlength - rq->decodedskip);
 
 out:
 	if (buff)
@@ -72,11 +71,11 @@ int z_erofs_decompress(struct z_erofs_decompress_req *rq)
 		if (rq->inputsize != EROFS_BLKSIZ)
 			return -EFSCORRUPTED;
 
-		DBG_BUGON(rq->outputsize > EROFS_BLKSIZ);
-		DBG_BUGON(rq->outputsize < rq->ofs_head);
+		DBG_BUGON(rq->decodedlength > EROFS_BLKSIZ);
+		DBG_BUGON(rq->decodedlength < rq->decodedskip);
 
-		memcpy(rq->out, rq->in + rq->ofs_head,
-		       rq->outputsize - rq->ofs_head);
+		memcpy(rq->out, rq->in + rq->decodedskip,
+		       rq->decodedlength - rq->decodedskip);
 		return 0;
 	}
 
fuse/decompress.h +6 -2
index 5a3e2356dc1b..7d436f18da86 100644
@@ -17,8 +17,12 @@ enum {
 struct z_erofs_decompress_req {
 	char *in, *out;
 
-	size_t ofs_head;
-	unsigned int inputsize, outputsize;
+	/*
+	 * initial decompressed bytes that need to be skipped
+	 * when finally copying to output buffer
+	 */
+	unsigned int decodedskip;
+	unsigned int inputsize, decodedlength;
 
 	/* indicate the algorithm will be used for decompression */
 	unsigned int alg;
fuse/read.c +2 -2
index 06ca08587e7c..46be5cc64a90 100644
@@ -136,9 +136,9 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 		ret = z_erofs_decompress(&(struct z_erofs_decompress_req) {
 					.in = raw,
 					.out = buffer + sum,
-					.ofs_head = ofs,
+					.decodedskip = ofs,
 					.inputsize = EROFS_BLKSIZ,
-					.outputsize = count,
+					.decodedlength = count,
 					.alg = algorithmformat,
 					.partial_decoding = partial
 					 });
Signature · 1 line
2.24.0
#16 lore raw

PATCH 10/12 erofs-utils: fuse: cleanup erofs_read_data_compression()

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
---
 fuse/read.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
fuse/read.c +16 -16
index 46be5cc64a90..f3aa628945e3 100644
@@ -80,10 +80,10 @@ finished:
 }
 
 size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
-		       size_t size, off_t offset)
+				   erofs_off_t size, erofs_off_t offset)
 {
 	int ret;
-	size_t end, count, ofs, sum = size;
+	erofs_off_t end, length, skip;
 	struct erofs_map_blocks map = {
 		.index = UINT_MAX,
 	};
@@ -91,8 +91,8 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 	unsigned int algorithmformat;
 	char raw[EROFS_BLKSIZ];
 
-	while (sum) {
-		end = offset + sum;
+	end = offset + size;
+	while (end > offset) {
 		map.m_la = end - 1;
 
 		ret = z_erofs_map_blocks_iter(vnode, &map);
@@ -100,7 +100,7 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 			return ret;
 
 		if (!(map.m_flags & EROFS_MAP_MAPPED)) {
-			sum -= map.m_llen;
+			end = map.m_la;
 			continue;
 		}
 
@@ -117,28 +117,28 @@ size_t erofs_read_data_compression(struct erofs_vnode *vnode, char *buffer,
 		 * larger than requested, and set up partial flag as well.
 		 */
 		if (end < map.m_la + map.m_llen) {
-			count = end - map.m_la;
+			length = end - map.m_la;
 			partial = true;
 		} else {
 			ASSERT(end == map.m_la + map_m_llen);
-			count = map.m_llen;
+			length = map.m_llen;
 			partial = !(map.m_flags & EROFS_MAP_FULL_MAPPED);
 		}
 
-		if ((off_t)map.m_la < offset) {
-			ofs = offset - map.m_la;
-			sum = 0;
+		if (map.m_la < offset) {
+			skip = offset - map.m_la;
+			end = offset;
 		} else {
-			ofs = 0;
-			sum -= count;
+			skip = 0;
+			end = map.m_la;
 		}
 
 		ret = z_erofs_decompress(&(struct z_erofs_decompress_req) {
 					.in = raw,
-					.out = buffer + sum,
-					.decodedskip = ofs,
-					.inputsize = EROFS_BLKSIZ,
-					.decodedlength = count,
+					.out = buffer + end - offset,
+					.decodedskip = skip,
+					.inputsize = map.m_plen,
+					.decodedlength = length,
 					.alg = algorithmformat,
 					.partial_decoding = partial
 					 });
Signature · 1 line
2.24.0
#17 lore raw

PATCH 11/12 erofs-utils: fuse: move up mpage in struct erofs_map_blocks

In reply to Gao Xiang via Linux-erofs
[ let's fold in to the original patch. ]
---
 include/erofs/internal.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/erofs/internal.h +2 -2
index 5807b67637c8..1637b6749411 100644
@@ -232,12 +232,12 @@ enum {
 #define EROFS_MAP_FULL_MAPPED	(1 << BH_FullMapped)
 
 struct erofs_map_blocks {
+	char mpage[EROFS_BLKSIZ];
+
 	erofs_off_t m_pa, m_la;
 	u64 m_plen, m_llen;
 
 	unsigned int m_flags;
-
-	char mpage[EROFS_BLKSIZ];
 	erofs_blk_t index;
 };
 
Signature · 1 line
2.24.0
#18 lore raw

PATCH 12/12 erofs-utils: fuse: fix up source headers

In reply to Gao Xiang via Linux-erofs

fix up weird paths and relicense zmap.c

[ let's fold in to the original patch. ]
---
 fuse/decompress.c | 5 +++--
 fuse/decompress.h | 3 ++-
 fuse/dentry.c     | 4 ++--
 fuse/dentry.h     | 4 ++--
 fuse/disk_io.c    | 4 ++--
 fuse/disk_io.h    | 4 ++--
 fuse/getattr.c    | 4 ++--
 fuse/getattr.h    | 4 ++--
 fuse/init.c       | 4 ++--
 fuse/init.h       | 4 ++--
 fuse/logging.c    | 4 ++--
 fuse/logging.h    | 4 ++--
 fuse/main.c       | 4 ++--
 fuse/namei.c      | 4 ++--
 fuse/namei.h      | 4 ++--
 fuse/open.c       | 4 ++--
 fuse/open.h       | 4 ++--
 fuse/read.c       | 5 +++--
 fuse/read.h       | 4 ++--
 fuse/readir.c     | 4 ++--
 fuse/readir.h     | 4 ++--
 fuse/zmap.c       | 7 ++++---
 22 files changed, 48 insertions(+), 44 deletions(-)
fuse/decompress.c +3 -2
index fc12852ac6b7..f2aa84146946 100644
@@ -1,9 +1,10 @@
-/* SPDX-License-Identifier: GPL-2.0+ */
+// SPDX-License-Identifier: GPL-2.0+
 /*
+ * erofs-utils/fuse/decompress.c
+ *
  * Copyright (C), 2008-2020, OPPO Mobile Comm Corp., Ltd.
  * Created by Huang Jianan <[email protected]>
  */
-
 #include <stdlib.h>
 #include <lz4.h>
 
fuse/decompress.h +2 -1
index 7d436f18da86..508aabab1664 100644
@@ -1,9 +1,10 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
+ * erofs-utils/fuse/decompress.h
+ *
  * Copyright (C), 2008-2020, OPPO Mobile Comm Corp., Ltd.
  * Created by Huang Jianan <[email protected]>
  */
-
 #ifndef __EROFS_DECOMPRESS_H
 #define __EROFS_DECOMPRESS_H
 
fuse/dentry.c +2 -2
index 27192ecfd32e..1ae37e3abc86 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\dentry.c
+ * erofs-utils/fuse/dentry.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "dentry.h"
 #include "erofs/internal.h"
 #include "logging.h"
fuse/dentry.h +2 -2
index f89c50646fb5..12f4cf6bafd9 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\dentry.h
+ * erofs-utils/fuse/dentry.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef _EROFS_DENTRY_H
 #define _EROFS_DENTRY_H
 
fuse/disk_io.c +2 -2
index 72d351b17806..3fc087699dc9 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\disk_io.c
+ * erofs-utils/fuse/disk_io.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #define _XOPEN_SOURCE 500
 #include "disk_io.h"
 
fuse/disk_io.h +2 -2
index 6b4bd3cce085..d2c3dd598bc0 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\disk_io.h
+ * erofs-utils/fuse/disk_io.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __DISK_IO_H
 #define __DISK_IO_H
 
fuse/getattr.c +2 -2
index d2134f486e19..e5200ebeef1a 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\getattr.c
+ * erofs-utils/fuse/getattr.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "getattr.h"
 
 #include <sys/types.h>
fuse/getattr.h +2 -2
index dbcff7c1a6e1..735529a91d5b 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\getattr.h
+ * erofs-utils/fuse/getattr.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __EROFS_GETATTR_H
 #define __EROFS_GETATTR_H
 
fuse/init.c +2 -2
index e9cc9f81d4c7..48125c5791fa 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\init.c
+ * erofs-utils/fuse/init.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "init.h"
 #include <string.h>
 #include <asm-generic/errno-base.h>
fuse/init.h +2 -2
index 3fc4eb548dda..405a92913b4a 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\init.h
+ * erofs-utils/fuse/init.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __EROFS_INIT_H
 #define __EROFS_INIT_H
 
fuse/logging.c +2 -2
index 2d1f1c77c2a4..192f546b94ec 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\logging.c
+ * erofs-utils/fuse/logging.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "logging.h"
 
 #include <stdio.h>
fuse/logging.h +2 -2
index 7aa2eda405db..3aa77ab08107 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\logging.h
+ * erofs-utils/fuse/logging.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __LOGGING_H
 #define __LOGGING_H
 
fuse/main.c +2 -2
index 884101374bcf..9c8169725fa7 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\main.c
+ * erofs-utils/fuse/main.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
fuse/namei.c +2 -2
index 0b71072027ce..c33af4b04b45 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\namei.c
+ * erofs-utils/fuse/namei.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "namei.h"
 #include <linux/kdev_t.h>
 #include <sys/types.h>
fuse/namei.h +2 -2
index 80e84d7220aa..1803a673daaf 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\inode.h
+ * erofs-utils/fuse/inode.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __INODE_H
 #define __INODE_H
 
fuse/open.c +2 -2
index 9d2edca54d23..c219d3870000 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\open.c
+ * erofs-utils/fuse/open.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "open.h"
 #include <asm-generic/errno-base.h>
 #include <fuse.h>
fuse/open.h +2 -2
index d02c1f752204..dfc8b3cdd515 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\open.h
+ * erofs-utils/fuse/open.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __EROFS_OPEN_H
 #define __EROFS_OPEN_H
 
fuse/read.c +3 -2
index f3aa628945e3..621f7d6ce6ea 100644
@@ -1,9 +1,10 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\read.c
+ * erofs-utils/fuse/read.c
+ *
  * Created by Li Guifu <[email protected]>
+ * Compression support by Huang Jianan <[email protected]>
  */
-
 #include "read.h"
 #include <errno.h>
 #include <linux/fs.h>
fuse/read.h +2 -2
index 89d4b4cd600c..e901c607dc91 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\read.h
+ * erofs-utils/fuse/read.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __EROFS_READ_H
 #define __EROFS_READ_H
 
fuse/readir.c +2 -2
index 46ceb1d90a7f..f3dd0c42c6e2 100644
@@ -1,9 +1,9 @@
 // SPDX-License-Identifier: GPL-2.0+
 /*
- * erofs-fuse\readir.c
+ * erofs-utils/fuse/readir.c
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #include "readir.h"
 #include <errno.h>
 #include <linux/fs.h>
fuse/readir.h +2 -2
index 21ab7a4e4e0b..ee2ab8bdd0f0 100644
@@ -1,9 +1,9 @@
 /* SPDX-License-Identifier: GPL-2.0+ */
 /*
- * erofs-fuse\readir.h
+ * erofs-utils/fuse/readir.h
+ *
  * Created by Li Guifu <[email protected]>
  */
-
 #ifndef __EROFS_READDIR_H
 #define __EROFS_READDIR_H
 
fuse/zmap.c +4 -3
index 022ca1b9e437..8ec0a7707fd6 100644
@@ -1,13 +1,14 @@
-// SPDX-License-Identifier: GPL-2.0-only
+// SPDX-License-Identifier: GPL-2.0+
 /*
- * Many parts of codes are copied from Linux kernel/fs/erofs.
+ * erofs-utils/fuse/zmap.c
+ *
+ * (a large amount of code was adapted from Linux kernel. )
  *
  * Copyright (C) 2018-2019 HUAWEI, Inc.
  *             https://www.huawei.com/
  * Created by Gao Xiang <[email protected]>
  * Modified by Huang Jianan <[email protected]>
  */
-
 #include "init.h"
 #include "disk_io.h"
 #include "logging.h"
Signature · 1 line
2.24.0